Stumping e-rater: challenging the validity of automated essay scoring

This report presents the findings of a research project funded by and carried out under the auspices of the Graduate Record Examinations Board Researchers are encouraged to express freely their professional judgment. Therefore, points of view or opinions stated in Graduate Record Examinations Board Reports do not necessarily represent official Graduate Record Examinations Board position or policy. The Graduate Record Examinations Board and Educational Testing Service are dedicated to the principle of equal opportunity, and their programs, services, and employment policies are guided by that principle. Abstract For this study, various writing experts were invited to "challenge" e-rater-an automated essay scorer that relies on natural language processing techniques-by composing essays in response to Graduate Record Examinations (GRE ®) Writing Assessment prompts with the intention of undermining its scoring capability. Specifically, using detailed information about e-rater's approach to essay scoring, writers tried to "trick" the computer-based system into assigning scores that were higher or lower than deserved. E-rater's automated scores on these " problem essays " were compared with scores given by two trained, human readers, and the difference between the scores constituted the standard for judging the extent to which e-rater was fooled. Challengers were differentially successful in writing problematic essays. Expert writers were more successful in tricking e-rater into assigning scores that were too high than in duping e-rater into awarding scores that were too low. The study provides information on ways in which e-rater, and perhaps other automated essay scoring systems, may fail to provide accurate evaluations, if used as the sole method of scoring in high-stakes assessments. The results suggest possible avenues for improving automated scoring methods.